Norm-preserving Orthogonal Permutation Linear Unit Activation Functions (OPLU)

Orthogonality preserving mappings on inner product C* -modules

Suppose that A is a C^*-algebra. We consider the class of A-linear mappins between two inner product A-modules such that for each two orthogonal vectors in the domain space their values are orthogonal in the target space. In this paper, we intend to determine A-linear mappings that preserve orthogonality. For this purpose, suppose that E and F are two inner product A-modules and A+ is the set o...

Linear preserving gd-majorization functions from Mn,m to Mn,k

Metric entropy and sparse linear approximation of lq-hulls for 0<q≤1

Consider `q-hulls, 0 < q ≤ 1, from a dictionary of M functions in L space for 1 ≤ p < ∞. Their precise metric entropy orders are derived. Sparse linear approximation bounds are obtained to characterize the number of terms needed to achieve accurate approximation of the best function in a `q-hull that is closest to a target function. Furthermore, in the special case of p = 2, it is shown that a ...

Approximating Orthogonal Matrices by Permutation Matrices

Motivated in part by a problem of combinatorial optimization and in part by analogies with quantum computations, we consider approximations of orthogonal matrices U by “non-commutative convex combinations”A of permutation matrices of the type A = P Aσσ, where σ are permutation matrices and Aσ are positive semidefinite n × n matrices summing up to the identity matrix. We prove that for every n× ...

DizzyRNN: Reparameterizing Recurrent Neural Networks for Norm-Preserving Backpropagation

The vanishing and exploding gradient problems are well-studied obstacles that make it difficult for recurrent neural networks to learn long-term time dependencies. We propose a reparameterization of standard recurrent neural networks to update linear transformations in a provably norm-preserving way through Givens rotations. Additionally, we use the absolute value function as an element-wise no...